Skip to content

test(devtools): plan snapshots for 32 small single-purpose workloads (#1278) - #1284

Merged
milindsrivastava1997 merged 2 commits into
mainfrom
test/1278-snapshot-workloads
Nov 18, 2026
Merged

milindsrivastava1997 merged 2 commits into
mainfrom
test/1278-snapshot-workloads

Conversation

@milindsrivastava1997

@milindsrivastava1997 milindsrivastava1997 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds 32 small, single-purpose workloads to sketch_plan_snapshots, each exercising one planner behavior. Closes #1278. Based on main 3f8fd295d (after #1417).

Review guide

  • crates/devtools/tests/sketch_plan_snapshots/manifest.json: the 32 new entries. Each covers says what the workload exercises (e.g. rollup-sum-chain: "sum by (job, host), by (job) and sum: rollups for sums"), not which plan is selected, so a plan change needs no rewording.
  • crates/devtools/tests/sketch_plan_snapshots.rs: one header paragraph.
  • workloads/*.json and <id>.json snapshots are generated or copied; skim them.
area ids
rollup across label sets rollup-quantile-nested-labels, rollup-quantile-disjoint-labels, rollup-sum-chain, stddev-nested-labels, avg-nested-labels, sql-rollup-quantile-same-window, sql-rollup-quantile-different-windows
top-k topk-5-and-10, topk-by-job-and-global, topk-5-and-10-accuracy, topk-rate-counter, topk-sum-count-over-time, topk-sum-count-over-time-accuracy
chains quantile-over-time-phase-0, chains-10m-5m-3m-3m, chains-10m-5m-3m-10s, chains-10m-5m-3m-10s-mvp, sum-over-time-two-intervals, quantile-phase-0-and-30s, quantile-7m-every-5m
over-time aggregates sum-avg-over-time-5m, sum-5m-avg-10m-over-time, mixed-quantiles
sketch choice distinct-over-time-1pct, distinct-over-time-5pct, entropy-over-time, quantile-two-targets, distinct-two-forms, entropy-and-distinct, quantile-exact, sign-unknown, sign-non-negative

All are on data (10k gauge series, 15 s samples), mostly every 1m at phase 0, under raw_retained (one under mvp). The two SQL workloads use workloads/sql-table-data.tables.json.

Behavior: before → after

rollup-sum-chain (sum by (job, host) (data), sum by (job) (data), sum (data)):

  • before: no snapshot.
  • after: the snapshot records the current plan, one Scan with by (job) and sum aggregated from the (job, host) result. A change to rollup shows up as a diff of this file.

Interfaces

None.

Algorithm changes

None.

Decisions and open questions

Verification

  • UPDATE_SNAPSHOTS=1 cargo test --release -p asap-devtools --test corpus_snapshots -- --ignored sketch_plans_match_snapshots on 3f8fd295d, then cargo test --release -p asap-devtools --test corpus_snapshots -- --include-ignored: 8 passed.
  • Snapshot changes: none of the 10 existing snapshots changed. The 32 new ones are added and record main's current plans. The second commit regenerates distinct-two-forms after fix(cost): print a summary build's operation counts to three decimals #1417 (116.19991733920547 ops → 116.2 ops, text only).
  • Reviewer confirms the recorded plans are an acceptable baseline: (for a person to fill in)

Follow-ups

🤖 Generated with Claude Code

https://claude.ai/code/session_014qhwq4GZoWwtfFWWr9CKkr

@milindsrivastava1997 milindsrivastava1997 changed the title test(devtools): sketch plan snapshots for 33 small single-purpose workloads (#1278) test(devtools): plan snapshots for 33 small single-purpose workloads (#1278) Nov 12, 2026
@zzylol
zzylol force-pushed the refactor/1207-tools branch 4 times, most recently from d744bd7 to bd3a333 Compare November 16, 2026 23:04
@zzylol

zzylol commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

The rebase onto main is clean, but regenerating on main changes 32 of the 33 new snapshots, and about 15 covers lines no longer describe the selected plan. For example:

Two workloads need changes beyond their covers lines:

  • quantile-over-time-no-phase uses fixed_interval, which main removed, so it no longer plans.
  • sign-unknown and sign-non-negative produce identical plans, so they pin nothing yet.

Could you rewrite those covers lines to match main's plans, or retune the workloads so the named behavior is selected?

The regenerated snapshots are on branch review/1284-regenerated-snapshots. It holds your commit rebased onto main 1760a69f5, plus one commit that regenerates the snapshots and drops quantile-over-time-no-phase. Feel free to cherry-pick or reset onto it.

A separate small issue: crates/cost/src/node/usage/mod.rs:1114 prints {update_ops} unrounded (for example "69.99993333806893 ops" in distinct-two-forms), which will churn the snapshots. Formatting it as {update_ops:.3} would fix it.

@milindsrivastava1997
milindsrivastava1997 force-pushed the test/1278-snapshot-workloads branch from e8c2135 to abe9d57 Compare November 18, 2026 16:00
@milindsrivastava1997
milindsrivastava1997 changed the base branch from refactor/1207-tools to main November 18, 2026 16:00
@milindsrivastava1997 milindsrivastava1997 changed the title test(devtools): plan snapshots for 33 small single-purpose workloads (#1278) test(devtools): plan snapshots for 32 small single-purpose workloads (#1278) Nov 18, 2026
@milindsrivastava1997

Copy link
Copy Markdown
Collaborator Author

Thanks. Rebased onto main 5400f93fd, with all snapshots regenerated there. Instead of tracking each selected plan, covers now says what the workload exercises (e.g. "sum by (job, host), by (job) and sum: rollups for sums"), so a cost-model change does not make it stale. Dropped quantile-over-time-no-phase. The sign pair stays: it exercises the declared sign, even though both currently select exact. The update_ops rounding is in #1417.

milindsrivastava1997 and others added 2 commits November 18, 2026 14:25
…kloads (#1278)

Each workload exercises one planner behavior: rollup across label sets,
top-k reuse, chain widths and sharing, avg as Sum / Count, and which
sketches an accuracy target or a declared value sign admits. Its
`covers` line says what the workload exercises, not which plan is
selected, so a plan change needs no rewording. Existing snapshots are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014qhwq4GZoWwtfFWWr9CKkr
The KMV build's operations per row print to three decimals
(116.19991733920547 -> 116.2). No plan changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014qhwq4GZoWwtfFWWr9CKkr
@milindsrivastava1997
milindsrivastava1997 force-pushed the test/1278-snapshot-workloads branch from abe9d57 to e317e20 Compare November 18, 2026 19:36
@milindsrivastava1997
milindsrivastava1997 merged commit eb0c68c into main Nov 18, 2026
4 checks passed
@milindsrivastava1997
milindsrivastava1997 deleted the test/1278-snapshot-workloads branch November 18, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add small single-purpose workloads to sketch_plan_snapshots

2 participants